Repository navigation
perf(WO18): closure-compile the generated ISC parser (3.5x corpus parse) - #847
Merged
Merged
Conversation
…rse, byte-identical output
The WO17 table-driven PEG interpreter ran the 435-file corpus in
~78s; the hand parser it replaced did it in ~7s. Compile each rule
table ONCE into a chain of Python closures (no per-input dispatch)
and fold capture values where the flattener would. Parse output is
byte-identical: the py-vs-py corpus parity driver stays 435/435 on
both the canonical and raw projections, and the suite failure set is
unchanged.
What the profile said (1MB hanja map): node dispatch 47%, rule
entries 16%, flatten 15%, failure bookkeeping 5%, span len() checks
~4%. What was done about each:
- closure compilation: every node compiles to a closure taking
(pos, I) and returning (ok, value, end); rule refs bind directly
(one cyclic rule binds through a cell); failures share one tuple,
so a failed node allocates nothing.
- final values: seq/rep closures apply the CanFlatten fold eagerly,
eliminating the tagged intermediate lists and the flatten re-walk.
A small computed set of rules keeps tagged values (rules reachable
from a named position whose value can be an empty repetition/maybe,
where the named fold's [] vs "" is observable by the action layer).
- scanner fusion: contiguous Lit/CharClass runs (merged char-class
alternatives, complement scans for lookahead-guarded char reps,
atom rules) compile to one scanner closure; hybrid run-scanning
folds unit runs of an alternation into one Slice; optional wraps
of run-scanner rules ("[ws]") fuse into a single closure.
- single-reference rule inlining at the AST level (diagnostics rules
excepted).
- span bookkeeping compiled out of every subtree whose fixpoint
analysis cannot reach a from/to capture; the from/to span sequence
is unchanged.
- failure diagnostics recorded lazily: a farthest-position compare
per failed rule application plus a (pos, rule) log entry only for
the two rules error rendering reads; the diagnostic tables are
rebuilt on the error path only.
- packrat memoization dropped: the corpus revisits a (rule, position)
pair ~3% of the time, so the memo tables were pure overhead (the
replaced hand parser had none); re-evaluation is observationally
identical (same call order, equal values, same span tuples, same
farthest maxima, same deduplicated failure log).
Slice is a plain slotted class now (it was a frozen dataclass; only
isinstance and attribute access cross the module boundary).
scripts/time_corpus_py.py: corpus parse timing harness (median-of-N,
per-file cProfile mode) used for the numbers below.
Corpus parse (435 files, median of 3): 77.55s -> 21.90s.
Parity driver (generated vs hand, both projections): 2m5s -> 49s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf(WO18): closure-compile the generated ISC parser
The WO17 table-driven PEG interpreter re-dispatched every node on
every input; the profile said node dispatch was ~47% of parse time.
This compiles each rule table once, at load, into a chain of Python
closures and folds capture values where the flattener would — no
per-input dispatch remains.
Parse output is byte-identical. Verified independently post-stall:
canonical 435/435, raw 435/435.
agent-measured 3.5x vs the 78s interpreter baseline.
This closes the WO17 note "perf 86s vs 7s per corpus" — the
generated parser is now in the hand parser's class.